System and method for providing customized connected device functionality and for operating a connected device via an alternate object

ABSTRACT

A system and method in a mobile device for controlling a connected device is disclosed. In one embodiment, the method includes receiving an image; processing the image to identify at least a first image attribute where the first image attribute includes an image feature or an image object; accessing a database of connected devices, each connected device in the database being identified by a linked image and being associated with device metadata describing at least a device type of the connected device; comparing the first image attribute with the linked images corresponding to the connected devices stored in the database; identifying a first connected device having a linked image matching the first image attribute; and in response to the identifying, providing a user interface configured to control the first connected device, the user interface including a control function corresponding to the device type of the first connected device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 15/961,443 filed Apr. 24, 2018, which claimspriority to U.S. Provisional Patent Application No. 62/634,670, entitledSYSTEM AND METHOD FOR PROVIDING CUSTOMIZED CONNECTED DEVICEFUNCTIONALITY AND FOR OPERATING A CONNECTED DEVICE VIA AN ALTERNATEOBJECT, filed Feb. 23, 2018, which is incorporated herein by referencefor all purposes.

FIELD

The present disclosure relates to a system and a method for providingcontrol of connected devices and, in particular, to a system and methodfor providing customized control to operate connected devices throughimage recognition.

BACKGROUND

With the advent of the Internet and the wide adoption of smartphones,the “smart home” concept or home automation can be realized whereappliances and devices in a home can be connected to the Internet and becapable of being monitored and controlled remotely. However,implementation of Internet controllable devices, referred to asInternet-of-Things (IoT) devices, in homes or offices remainschallenging in various aspects.

For example, an environment with multiple IoT devices or connecteddevices being installed in a closed proximity presents challenges to auser. For instance, it is sometimes difficult to identify which IoTdevice is the one referenced inside a particular application (App). Thiscould lead to turning ON or OFF the incorrect IoT deviceunintentionally. Alternately, a user may opt for voice commands.However, when there are multiple devices, it is difficult for the userto remember the designated name of each IoT device when it wasoriginally programmed. Also, voice command is only good for simpleinteractions, such as short commands of On, Off, Higher, and Lower.Voice commands have limitations in providing general control.

Conventional methods for controlling or operating IoT devices usesapplications (Apps) that are installed on a mobile device. As each IoTdevice may have its own associated App, a user may have to navigatearound to locate a specific App for a specific IoT device. Furthermore,the user interface for each App are often designed differently. Withmultiple IoT devices and different styled Apps, a user often finds itconfusing and difficult to master the navigation of each App forcontrolling the multiple IoT devices.

SUMMARY

The present disclosure discloses a device and method for controlling aconnected device in a mobile device, substantially as shown in and/ordescribed below, for example in connection with at least one of thefigures, as set forth more completely in the claims.

These and other advantages, aspects and novel features of the presentdisclosure, as well as details of an illustrated embodiment thereof,will be more fully understood from the following description anddrawings.

In embodiments of the present disclosure, a method in a mobile devicefor controlling a connected device includes receiving, at the mobiledevice, an image; processing, at the mobile device, the image toidentify at least a first image attribute, the first image attributeincluding an image feature or an image object; accessing, by the mobiledevice, a database of connected devices, each connected device in thedatabase being identified by a linked image and being associated withdevice metadata describing at least a device type of the connecteddevice; comparing the first image attribute with the linked imagescorresponding to the connected devices stored in the database;identifying a first connected device having a linked image matching thefirst image attribute; and in response to the identifying, providing, atthe mobile device, a user interface configured to control the firstconnected device, the user interface including a control functioncorresponding to the device type of the first connected device.

In embodiments of the present disclosure, a system for controlling aconnected device includes an imaging sensing device configured toreceive an image; a processor; a communication interface; a display; anda memory coupled with the processor, wherein the memory is configured toprovide the processor with instructions which when executed cause theprocessor to: receive an image; process the image to identify at least afirst image attribute, the first image attribute including an imagefeature or an image object; access a database of connected devices, eachconnected device in the database being identified by a linked image andbeing associated with device metadata describing at least a device typeof the connected device; compare the first image attribute with thelinked images corresponding to the connected devices stored in thedatabase; identify a first connected device having a linked imagematching the first image attribute; and in response to the identifying,provide a user interface configured to control the first connecteddevice, the user interface including a control function corresponding tothe device type of the first connected device.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure are disclosed in thefollowing detailed description and the accompanying drawings.

FIG. 1 illustrates an environment in which the image-based userinteraction system of the present disclosure can be implemented.

FIG. 2 is a schematic diagram of a mobile device in which theimage-based user interaction system can be implemented in some examples.

FIG. 3 is a schematic diagram of a hub which can be implemented in theenvironment 10 of connected devices in some examples.

FIG. 4 is a flowchart illustrating a method to on-board a connecteddevice in some embodiments.

FIG. 5 is a flowchart illustrating a method to on-board a dummy deviceand link the dummy device in some embodiments.

FIG. 6 is a flowchart illustrating a method to retrieve and operate aconnected device in some embodiments.

FIG. 7, which includes FIGS. 7(a) and 7(b), illustrates exampleapplications of the image-based user interaction system in someembodiments.

FIG. 8, which includes FIGS. 8(a) and 8(b), illustrates exampleapplications of the image-based user interaction system in someembodiments.

FIG. 9 is a flowchart illustrating a facial recognition retrieval methodin some embodiments.

FIG. 10 illustrates on example of the operation of the facialrecognition retrieval method.

FIG. 11 is a flowchart illustrating a geolocation retrieval method insome embodiments.

FIG. 12 illustrates on example of the operation of the geolocationretrieval method.

DETAILED DESCRIPTION

The present disclosure can be implemented in numerous ways, including asa process; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a hardware processor or a processor device configuredto execute instructions stored on and/or provided by a memory coupled tothe processor. In this specification, these implementations, or anyother form that the present disclosure may take, may be referred to astechniques. In general, the order of the steps of disclosed processesmay be altered within the scope of the present disclosure. Unless statedotherwise, a component such as a processor or a memory described asbeing configured to perform a task may be implemented as a generalcomponent that is temporarily configured to perform the task at a giventime or a specific component that is manufactured to perform the task.As used herein, the term ‘processor’ refers to one or more devices,circuits, and/or processing cores configured to process data, such ascomputer program instructions.

A detailed description of one or more embodiments of the presentdisclosure is provided below along with accompanying figures thatillustrate the principles of the present disclosure. The presentdisclosure is described in connection with such embodiments, but thepresent disclosure is not limited to any embodiment. The scope of thepresent disclosure is limited only by the claims and the presentdisclosure encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of the presentdisclosure. These details are provided for the purpose of example andthe present disclosure may be practiced according to the claims withoutsome or all of these specific details. For the purpose of clarity,technical material that is known in the technical fields related to thepresent disclosure has not been described in detail so that the presentdisclosure is not unnecessarily obscured.

According to embodiments of the present disclosure, a system and methodin a mobile device for controlling a connected device processes an imageto identify the connected device to be controlled and provides a userinterface including control functions associated with the identifiedconnected device. In some embodiments, the image includes the desiredconnected device. In other embodiments, the image is of a dummy devicethat is not the connected device but is linked to the connected device.In another embodiment, the image may be a human face identifying a user.In yet another embodiment, the image may be a location in which theconnected device is located. The system and method of the presentdisclosure provides an easy and quick way for users to interact withconnected devices by using a mobile device to take an image of theconnected device or a linked dummy device. The control functions for thespecific connected device desired by the user can be provided instantlyon the mobile device, without requiring the user to search for theconnected device on in an application or to search for the particularapplication associated with the connected device.

In some embodiments, the image-based user interaction system and methodof the present disclosure uses machine learning and computer vision torecognize image features or an image object from an image andautomatically display the corresponding device user interface (UI) forcontrolling the connected device linked to the image features or object.In particular, a unified application (App) can be used to control alltypes of connected devices, improving the accessibility and ease of useof all types of connected devices.

In the present description, connected devices refer to devices that havethe capability to be connected to a data network and havefunctionalities that can be controlled remotely through the datanetwork. The data network can include a cellular data network, a Wi-Fidata network, or other types of data communication network, such asBluetooth, Zigbee, Thread. In the present description, connected devicesare sometimes referred to as Internet-of-Things (IoT) devices and aregenerally devices applied in home or office automation applications. Insome cases, a connected device may be connected to the data networkthrough a hub. In operation, the hub functions as a bridge to enable adevice to be connected to the data network. In this manner, a connecteddevice can be controlled and operated through the hub without requiringthe connected device to implement full network communication capability.

In the present description, a dummy device refers a device that is notthe actual connected device for which control and operation is desired.A dummy device can be any object and can be device that may or may nothave the capability to be connected to a data network. When used as adummy device, the ability of the device to be connected to a datanetwork is irrelevant as the user interaction system is not attemptingto control or operate the dummy device itself. In some examples, thedummy device can be an everyday object, such as a bicycle, a cup, atoaster. The dummy device may be an unconnected device that does nothave the capability to be connected to a data network. Alternately, thedummy device can be a device that has the capability to be connected toa data network but is offline or not registered with the system of thepresent disclosure and for which control and operation through the userinteraction system is not sought.

In the present description, an application, or an App, refers tosoftware components that is configured to monitor, control and automateconnected devices installed in an environment or at a location. Inembodiments of the present disclosure, the image-based user interactionsystem enables a user to operate or control all of the user's connecteddevices using a single control application (control App). The user nolonger needs to rely on device-specific App for each connected device.In embodiments of the present disclosure, the image-based userinteraction system uses a camera to recognize the connected device orthe linked dummy device, a human face, or the environment where theconnected device is located, and build a control link between theconnected device and the control application either over a cloud datanetwork or a local data network. In this manner, the image-based userinteraction system leverages computer vision technologies to create auser interaction that maps physical items to virtual user interface. Theuser interaction system simplifies the user experience when interactingwith multiple connected devices on different technologies.

The image-based user interaction system is particularly advantageouswhen applied in an environment where a wide range of connected devicesis deployed within an area and the user is looking to control eachconnected device in a different fashion. In some embodiments, theimage-based user interaction system implements the following functions.

In a first example, the user interaction system uses the control App toidentify a connected device by visual information of the connecteddevice and provides the user with a user interface (UI) for deviceoperation designated for the connected device. Alternately, the controlApp may identify a dummy device that is linked to the connected deviceand provides the user with a user interface (UI) for device operationdesignated for the connected device.

In a second example, the user interaction system uses the control App toidentify the user, such as based on facial recognition, and provides theuser with a list of connected devices based on the user's previousbehaviors. For example, the list can be the user's most frequently usedor most recently used connected devices.

In a third example, the user interaction system uses the control App todetermine the location of the user, such as using geolocation, andprovides the user with a list of connected devices nearby or in thevicinity of the determined location. For example, the list can be theuser's most frequently used or most recently used connected devicesassociated with the current location.

The image-based user interaction system of the present disclosureimplements many advantageous features. First, the image-based userinteraction system uses computer vision and machine learning to performthe initial device on-boarding using a digital camera. The systemfurther retrieves a particular device from the set of on-boarded devicesusing the digital camera and computer vision. The image-based userinteraction system enables a user to easily identify and control his/herconnected device, representing a marked improvement over voice enablecontrol methods. In this manner, there is no need to remember theassigned name for each connected device in order to control or operatethe device.

Second, the image-based user interaction system learns user preference,such as the most commonly used connected devices. Alternately, thesystem associates the most commonly used connected devices to user'sfacial image or to a geolocation. The user can save a lot of time byeliminating the need to navigate among different devices to search forthe desired connected device to control. A user can quickly selecthis/her own most accessed connected devices based on facial recognitionor geolocation.

Third, based on the image being scanned, the system links a matchingvirtual control user interface to an actual physical device controlinterface. After the link is established between physical device andvirtual UI, the user can use the control App to control the physicaldevice using the virtual control UI. The image-based user interactionsystem enables ease of use as the user no longer has to search for theright control UI for each particular connected device.

FIG. 1 illustrates an environment in which the image-based userinteraction system of the present disclosure can be implemented.Referring to FIG. 1, multiple connected devices 20-23 are installed inan environment 10, which can be a home, an office, a manufacturing site,or an outdoor location. In the present illustration, the connecteddevices include a thermostat 20, a Bluetooth speaker 21, a lamp 22 and afan 23. The connected devices may communicate with a data network, suchas data network 16, to be monitored, controlled and operated. The datanetwork 16 can include a cellular data network, a Wi-Fi data network, orother types of data communication network. In some cases, the connecteddevice, such as lamp 22 and fan 23, may be connected to a hub 14 to becontrolled or operated. The connected devices may communicate with thehub 14 using short range or low power communication protocols, such asBluetooth, Zigbee, and Thread. In some examples, a web-based centralserver, referred to as cloud server 18, may provide remote monitoringand control of the connected devices 20-23. In the present illustration,cloud server 18 is representative of one or more web-based serversconfigured for controlling respective connected devices.

A user may control and operate the connected devices 20-23 using amobile device 12, such as a smartphone or a tablet. The mobile device 12may communicate with the connected devices 20-23 through the datanetwork 16 or through the hub 14, using a wireless or cellularconnection. In one example, a user may access the services provided bycloud server 18 to remotely control and operate the connected devices20-23.

Using the conventional control scheme, the user may need to accessdifferent Apps for controlling each connected device. For example, thethermostat 20 may have its own associated App while the home assistant21 may use another App for its control and operation. Searching for andmanaging all the different Apps for different connected devices may becumbersome for the user. Furthermore, each App may have a different userinterface design, requiring the user to navigate different userinterfaces to figure out the control function.

Embodiments of the present disclosure provide an image-based userinteraction system that enhances the user experience by enabling a userto interact with connected devices using an image-based device accessscheme and using a unified control application (control App), as will bedescribed in more detail below.

FIG. 2 is a schematic diagram of a mobile device in which theimage-based user interaction system can be implemented in some examples.Referring to FIG. 2, a mobile device 12 includes a digital camera 52 forcapturing a digital image, a memory 54 for storing data, a communicationinterface 56 for supporting cellular and/or wireless communication, anda display 58 providing the user interface. The mobile device 12 includesa processor 50, which can be a micro-controller or a micro-processor,for controlling the operation of the mobile device. In some embodiments,the processor 50 may implement a computer vision sub-system.

FIG. 3 is a schematic diagram of a hub which can be implemented in theenvironment 10 of connected devices in some examples. Referring to FIG.3, the hub 14 may be deployed in an environment to function as a bridgefor communicating with and controlled connected devices. The hub 14includes a network interface 60 for communicating with a data network,such as data network 16, to enable communication with the cloud server18. The hub 14 may further include a device interface 62 forcommunicating with connected devices. In some examples, the networkinterface 60 may be a wired interface or a wireless interface, such asWi-Fi interface. Meanwhile, the device interface 62 may be a wirelessinterface, implementing Wi-Fi communication and/or other short-range,low power wireless communication, such as Bluetooth or Zigbee or Thread.The hub 14 includes a processor 64 controlling the operation of the huband a memory 66 storing data.

The image-based user interface system and method provides controlfunctionalities for connected devices by assessing a database ofconnected devices. The connected devices are on-boarded to the databasethrough an on-boarding process in accordance with the presentdisclosure.

In the present description, the connected devices are assumed to havecompleted the device registration process so that the connected devicesare connected to existing data networks and are performing theirintended functions. In some cases, the connected devices may havecompleted a device registration process with a cloud server associatedwith a manufacturer of the connected device. In other cases, theconnected devices may have completed a device registration process andare connected to a hub local to the connected devices where the hubfacilitates the communication with a remote cloud server to support theintended functionality of the connected device. For example, a smartlight switch from ACME, Inc. will have completed a device registrationprocess with the ACME cloud server to enable the smart light switch tobe operated by an ACME application on the mobile device of the user.Alternately, the smart light switch may have completed a deviceregistration process through a local hub and is configured tocommunicate with the hub acting as a bridge to communicate with the ACMEcloud server. The device registration process for a connected device toconnect to the existing data networks to provide its intendedfunctionality is known in the arts and is not part of the image-baseduser interaction system and method. In particular, implementation of theimage-based user interaction system and method assumes that theconnected devices have been registered with their respective cloudservers of the manufacturers or providers and are operating to providetheir intended functionality. The image-based user interaction systemand method of the present disclosure uses visual information to provideenhanced control functions for the connected devices.

On-Boarding Method for Connected Devices

FIG. 4 is a flowchart illustrating a method to on-board a connecteddevice in some embodiments. Referring to FIG. 4, a connected deviceon-boarding method 150 may be implemented in a mobile device andexecuted by the control App in embodiments of the present disclosure. Atstep 152, the method 150 initiates the control App and detects an imagein the camera field of view where the image contains the connecteddevice to be on-boarded. It is instructive to note that the on-boardingmethod does not necessarily require the camera to capture or snap theimage. In some embodiments, it is only necessary that an image ispresent in the camera field of view.

At step 154, the method 150 processes the image to identify imagefeatures and/or a recognized image object. In one embodiment, thecomputer vision sub-system in the processor of the mobile device may beinitiated to scan the image content in the camera field of view (FOV).The computer vision sub-system extracts image features from the field ofview, which can be used as a unique representation of the connecteddevice and the device's surrounding environment. In some embodiments,the computer vision sub-system may also run a vision-based objectrecognition method to recognize the connected device to be on-boarded.In the present description, image features refer to derived values thatare informative and descriptive of the image, such as edges and contoursin the digital image. In the present description, image objects refer toinstances of semantic objects in the digital image, such as a vase, alamp or a human. In the following description, the term “imageattributes” is sometimes used to refer collectively to image features orimage object of an image.

At step 156, the method 150 receives device metadata associated with theconnected device. The device metadata may include device informationsuch as the device type, the brand, the model, the operational propertyand other attributes of the connected device. The device metadatainformation is provided to enable the user to interact with theconnected device later. Using the device metadata information, the userinteraction system of the present disclosure is then able to assign thecorresponding user interface, to determine the functional properties andthe communication protocol of the connected device.

At step 158, the method 150 registers the connected device and storesthe connected device in the database of connected devices. In thepresent description, registering a device refers to adding or enteringthe device into the image-based user interaction system of the presentdisclosure using information provided about the device. In particular,the connected device is stored in the database identified by the imagefeatures or recognized image object as the linked image. The associateddevice metadata is also stored with the connected device in thedatabase.

In embodiments of the present disclosure, the database of connecteddevices may be stored in the mobile device. Alternately, the databasemay be stored in the hub in the environment of connected devices and themobile device access the hub, such as hub 14 in FIG. 1, to access thedatabase. In yet another embodiment, the database may be stored in aremote server, such as the cloud server 18 of FIG. 1, and the mobiledevice accesses the cloud server to access the database.

Using the connected device on-boarding method 150, one or more connecteddevices may be on-boarded or registered with the image-based userinteraction system. In embodiments of the present disclosure, theimage-based user interaction system implements a first control schemereferred to as “See-it, Control-it” scheme. Under the See-it, Control-itcontrol scheme, the image-based user interaction system enables a mobiledevice to control a connected device by aiming the camera of the mobiledevice to the connected device itself. That is, presenting an image of aconnected device enables control of that connected device.

In some cases, the user may wish to control a connected device that isnot necessarily in view of the user. In other cases, connected devicescannot be reached via voice command or touch command or by an IR remotedue to location challenge. In embodiments of the present disclosure, theimage-based user interaction system implements a second control schemereferred to as “See-this, Control-that” scheme. Under the See-this,Control-that control scheme, the image-based user interaction systemenables a connected device to be linked to a dummy device. The dummydevice is used as a substitute object or an alternate object torepresent the connected device. The mobile device can then control theconnected device by aiming the camera of the mobile device to the linkeddummy device. That is, presenting an image of a dummy device enablescontrol of a linked connected device. In other words, the mobile devicemay be used to scan an image of the dummy device and the user interfacefor controlling the linked connected device will be presented.

As described above, a dummy device refers a device that is not theactual connected device for which control and operation is desired. Adummy device, or a substitute device, can be an object or an item of anyshape, size or form as long as it can be recognized by the computervision sub-system. For example, the dummy device can be a pattern, apicture, a human face, a number, a hand gesture (e.g. thumbs up), or aphysical object, such as a kitchen appliance. In some embodiments, thedummy device can be any animated objects (e.g., a dog or a cat) or anyinanimate objects (e.g., a cup or a book). The dummy device does nothave to be connected to a data network.

To facilitate the See-this, Control-that control scheme, a dummy deviceon-boarding and linking method is used to on-board the dummy device andlink the dummy device to the actual connected device it represents.

On-Boarding and Linking Method for Dummy Devices

FIG. 5 is a flowchart illustrating a method to on-board a dummy deviceand link the dummy device in some embodiments. Referring to FIG. 5, adummy device on-boarding method 180 may be implemented in a mobiledevice and executed by the control App in embodiments of the presentdisclosure. In some embodiments, the dummy device is registered andstored in the database of connected devices in the same manner as aconnected device, but the dummy device stores a link to the linkedconnected device in the database structure of the dummy device.

At step 182, the method 180 initiates the control App and detects animage in the camera field of view where the image contains the dummydevice to be on-boarded. It is instructive to note that the on-boardingmethod does not necessarily require the camera to capture or snap theimage. In some embodiments, it is only necessary that an image ispresent in the camera field of view.

At step 184, the method 180 processes the image to identify the imageattributes of the image, that is, to identify the image features and/ora recognized image object in the image. In one embodiment, the computervision sub-system in the processor of the mobile device may be initiatedto scan the image content in the camera field of view. The computervision sub-system extracts image features from the field of view, whichcan be used as a unique representation of the dummy device and thedevice's surrounding environment. In some embodiments, the computervision sub-system may also run a vision-based object recognition methodto recognize the dummy device to be on-boarded.

At step 186, the method 180 receives device metadata associated with thedummy device. In the case of the dummy device, the device metadata mayinclude a device identifier, such as a device name.

At step 188, the method 180 registers the dummy device and stores thedummy device in the database of connected devices. In particular, thedummy device is identified by the image features or recognized imageobject as the linked image and is stored with the associated devicemetadata.

At step 190, the method 180 provides a list of connected devices fromthe database of connected device. For example, the list can be displayedon a user interface of the mobile device. At step 192, the method 180receives a selection of a connected device. The selected connecteddevice is to be linked to the dummy device registered previously.

At step 194, the method 180 stores linking data of the dummy device inthe database linking the dummy device to the selected connected device.The linking data may be stored in the data structure of the dummy devicein the database. In some embodiments, the linking data may includeinformation of the selected connected device, such as the deviceidentifier of the connected device.

Using the dummy device on-boarding and linking method 180, one or moredummy devices may be on-boarded and may be linked to connected devicesto facilitate the See-this, control-that control scheme, as will bedescribed in more detail below.

Connected Device Retrieval Method

With the connected devices and dummy devices on-boarded, the image-baseduser interaction system may be deployed to enable simple and easyretrieval and control of multiple connected devices in an environment.FIG. 6 is a flowchart illustrating a method to retrieve and operate aconnected device in some embodiments. Referring to FIG. 6, a connecteddevice retrieval method 100 may be implemented in a mobile device andexecuted by the control App in embodiments of the present disclosure.

At step 102, the method 100 initiates the control App and detects animage in the camera field of view. The image may contain a connecteddevice or a dummy device. It is instructive to note that the retrievalmethod does not necessarily require the camera to capture or snap theimage. In some embodiments, it is only necessary that an image ispresent in the camera field of view.

At step 104, the method 100 processes the image to identify the imageattributes of the image, that is, to identify the image features and/ora recognized image object in the image. In one embodiment, the computervision sub-system in the processor of the mobile device may be initiatedto scan the image content in the camera field of view. The computervision sub-system extracts image features from the field of view. Insome embodiments, the computer vision sub-system may also run avision-based object recognition method to recognize an image object.

At step 106, the method 100 accesses the database of connected devices.At 108, the method 100 compares the extracted image features orrecognized image object with linked images in the database. The compareoperation can be performed on the mobile device, such as when thedatabase is stored on the mobile device. The compare operation can alsobe performed at the hub or on the cloud server, in the cases where thedatabase is stored on the hub or at the cloud server.

At step 110, the method 100 identifies a connected device with thematching linked image. In one embodiment, the extracted image featuresor recognized image object may match a linked image of the connecteddevice itself. In another embodiment, the extracted image features orrecognized image object may match a linked image of the dummy device. Inthat case, the connected device linked to the dummy device is identifiedas the connected device.

At step 112, the method 100 provides a user interface designated forcontrolling the connected device identified by the linked image. Forexample, the user interface may be provided on the display of the mobiledevice. In particular, the method 100 selects the designated userinterface based on the device metadata associated with the connecteddevice. The device metadata contains information about the connecteddevice, such as the device type, the brand, the model and the propertyof the device. Using the device metadata, the method 100 provides thedesignated user interface for the connected device. For example, whenthe connected device is a lamp, the user interface may provide thecontrol functions of On/Off and brightness control. In another example,when the connected device is a speaker, the user interface may providethe control functions of On/Off and volume control.

As thus configured, the method 100 provides control functions forvarious types of connected devices through a unified control App.Importantly, the same user interface is presented for the connecteddevices of the same device type. For example, the environment mayinclude multiple lamps as connected devices, each lamp may be of adifferent brand, different manufacturers and thus associated withdifferent native control interface. Instead of having the user navigatethe different control interfaces for each lamp, the method 100 of thepresent disclosure provides the same user control interface in thecontrol App for all types of lamps as connected device. A user is thusable to control all of the lamps in his/her environment using the sameuser interface. The method 100 improves the ease of use and convenienceto the user.

In embodiments of the present disclosure, for the purpose ofvision-based on-boarding and retrieving process, the image processingoperation leverages computer vision and machine learning to identifyimage features and recognize image object. In some embodiments, variousfeature extraction methods can be applied, such as Scale-InvariantFeature Transform (SIFT), Speeded-Up Robust Features (SURF), Histogramof oriented gradients (HOG), as well as deep neural network flavor offeature extractors, such as VGGnet, ResNet, etc. In other embodiments,various object recognition methods can be applied, such as Bag of Worldbased classification, as well as deep neural network flavor of objectrecognition, such as fast-RNN, MobileNet, and Yolo.

In embodiments of the present disclosure, the methods of the presentdisclosure may process the image by identifying image features and imageobjects in the foreground of the image. In some cases, foreground andbackground image features can be distinguished using depth estimation,using an attention network to detect significant object, or selectingthe biggest object as the foreground object. In other embodiments, themethods of the present disclosure may process the image by identifyingimage features and image objects in the foreground as well as in thebackground. That is, an object, whether a connected device or a dummydevice, may be identified by just the object itself in the image or bythe object and the surrounding environment in the image. Accordingly,the on-boarding and retrieval methods of the present disclosure may beapplied to identify a connected device located in a particular room andto differentiate another identical connected device that may be locatedin a different room.

For example, the on-boarding method may process an image to identify alamp. Alternately, the on-boarding method may process an image toidentify a lamp in the living room. Subsequently, the retrieval methodmay identify a lamp in the image and retrieve the lamp from the databaseof connected devices. Alternately, the retrieval method may identify alamp in the living room in the image and retrieve the connected devicebeing a lamp located in the living room from the database of connecteddevices, instead of the same lamp located in the bedroom.

FIG. 7, which includes FIGS. 7(a) and 7(b), illustrates exampleapplications of the image-based user interaction system in someembodiments. FIG. 7(a) illustrates an example of the See-it, Control-itcontrol scheme in some embodiments. Referring to FIG. 7(a), a userwishes to control a lamp. It is assumed that the lamp has beenpreviously on-boarded as a connected device and stored in the database.The user initiates the control App on her mobile device and points thecamera of the mobile device at the light. The control App recognizesthat the object is a lamp and identifies the lamp as an on-boardedconnected device in the system. Immediately, the control App providesthe lighting control user interface on the display of the mobile device.The user is then able to control the lamp within control App using thelighting control user interface, as shown in FIG. 7(a). In operation,the user does not need to continue to point the camera at the lamp whilenavigating the lighting control user interface. In operation, the cameraonly needs to point to the lamp for a sufficient amount of time for thecontrol App is locked onto the image of the connected device (the lamp).After that, the control App no longer needs the image to continue theoperation.

FIG. 7(b) illustrates an example of multiple connected devices in thesame view. For example, the image-based user interaction system of thepresent disclosure may be used to control industrial machinery. In theevent that multiple on-boarded devices are detected in the same image,the control App provides a user interface to allow the user to selectwhich connected device to be controlled.

In embodiments of the present disclosure, the image-based userinteraction system may capture multiple connected devices at the sametime within the camera field of view, either during the on-boardingprocess or during the retrieval process. In some embodiments, theon-boarding method or the retrieval method will query the user to selectan area within the field of view for the computer vision system to focuson. The on-boarding or retrieval method then continues by processing theimage in the selected area for visual features or image object.

FIG. 8, which includes FIGS. 8(a) and 8(b), illustrates exampleapplications of the image-based user interaction system in someembodiments. In particular, FIGS. 8(a) and 8(b) illustrate examples ofthe See-this, Control-that control scheme in some embodiments. Under theSee-this, Control-that control scheme, a user on-boards a dummy deviceand then links the dummy device to the connected device that the userwants to control. In this manner, even if the actual connected device isin a remote location or not in the field of view of the user, the usercan still operate the connected device via the dummy device.

FIG. 8(a) illustrates pictorially the See-this, Control-that controlscheme in one example. Referring to FIG. 8(a), a connected lamp(connected device A) is on-boarded onto the database and a coffee mug(dummy device B) is also on-boarded. The coffee mug is linked to theconnected lamp. Thereafter, when the user scans the coffee mug, theimage-based user interaction system identifies the coffee mug andretrieves the user interface for controlling the lamp. In this manner,the user is able to operate the lamp via the coffee mug.

FIG. 8(b) illustrates another example of applying the See-this,Control-that control scheme in one example. In this example, a userwants to control a television (not shown) in the living room, the userinitiates the control App and points the mobile device at the coffee muglocated in the kitchen. It is assumed that both the television and thecoffee mug have both been previously on-boarded and linked. As a resultof the image of the coffee mug being presented to the control App, theApp recognizes the coffee mug that was previously on-boarded and linkedto the television. Immediately, the control App provides televisioncontrol user interface on the display of the mobile device. For example,the television control user interface may provide a volume controlbutton and a program channel selection button. In this manner, the useris then able to control the television using control App on the mobiledevice using the television control user interface—even though thetelevision is in a different room. In operation, the user does not needto continue to point the camera at the coffee mug while navigating thetelevision control user interface.

In general, under the See-this, Control-that control scheme, the useron-boards both device A (connected device) and device B (dummy device)and then link device B to device A. Subsequently, the user can scandevice B and the control App will displays device A's control userinterface on the display. The user can then control device A within thecontrol App. In some embodiments, the user may use the control App toremotely add or delete hidden digital content associated with the dummydevice.

The See-this, Control-that control scheme provides many advantagesfeatures. Using a dummy device, the image-based user interaction systemenables the user to control a connected device even if the connecteddevice is out of reach by voice command, by touch or by IR remote. Theimage-based user interaction system enables the user to save time whensearching for the right control interface matching to each particularconnected device. In operation, the user may trigger the control userinterface using the connected device or a dummy device linked to theconnected device.

According to embodiments of the present disclosure, the image-based userinteraction system enables control of a retrieved connected device usinga user interface designated for the device type of the connected device.The operation of the connected device can be carried out as follows. Inone example, the user gives a command (e.g. button press) on the userinterface of the control application to turn on a connected lamp. Thecontrol application receives the button press event and converts theevent to a standardized data-interchange format, such as JSON and xml.

Then, the transmitter sub-system in the communication interface of themobile device sends out the command in a suitable format either to agateway (a hub) that helps to route the signal or to the end device(e.g. a lamp). The transition can be done either through a wiredconnection (e.g. USB, UART, I2C, etc.), or through a wireless connection(e.g. Wi-Fi, Bluetooth, Zigbee, Thread).

In one embodiment, the gateway (hub) receives the command data androutes the command data to a particular end device according to thedevice metadata carried by the command data. In another embodiment, theend device receives the command data directly according to the devicemetadata carried by the command data.

The receiver sub-system on communication interface of the end devicelistens to the incoming command and receives the command designated tothe end device. The command parsing sub-system of the end device parsesthe incoming command data and converts it to an action that can beexecuted by the end device. The user's command is finally executed bythe end device.

Retrieving Connected Devices by User Facial Recognition

Accordingly, to another aspect of the present disclosure, theimage-based user interaction system is configured to provide the userwith a list of device control user interfaces based on a face scan ofthe user. In some embodiments, the system provides a list of mostfrequently used device control user interfaces associated with the user.In other embodiments, the system provides a list of most recently useddevice control user interfaces associated with the user. In otherembodiments, the system provides a list of device control userinterfaces predefined by the user.

FIG. 9 is a flowchart illustrating a facial recognition retrieval methodin some embodiments. Referring to FIG. 9, a facial recognition retrievalmethod 200 may be implemented in a mobile device and executed by thecontrol App in embodiments of the present disclosure.

At step 202, the method 200 initiates the control App and detects animage in the camera field of view. The image may contain a human face.It is instructive to note that the retrieval method does not necessarilyrequire the camera to capture or snap the image. In some embodiments, itis only necessary that an image is present in the camera field of view.

At step 204, the method 200 processes the image to identify a human faceas the recognized image object. In one embodiment, a face recognitionsub-system in the processor of the mobile device may be initiated toscan the face image in the camera field of view and to recognize theuser associated with the face image.

At step 206, the method 200 access a database of registered users. Eachregistered user being identified by a linked face image and beingassociated with usage data for one or more connected devices. At step208, the method 200 identifies a registered user in the database bymatching the face image identified in the image to the linked face imageof the registered user.

At step 210, in response to identifying the registered user, the method200 provides a list of connected devices associated with the user. Thelist can be a list of most frequently used connected devices, or a listof most recently used connected devices, or a predefined list ofconnected devices.

At step 212, the method 200 receives an input selection selecting aconnected device from the list. At step 214, the method 200 provides auser interface designated for controlling the selected connected device.

FIG. 10 illustrates on example of the operation of the facialrecognition retrieval method. Referring to FIG. 10, the control App onthe mobile device recognizes the face image presented as a registereduser and provides the user with a list of connected devices—a lamp, atelevision and a game console—on the control App. The user can thenquickly select a connected device to operate.

In some embodiments, a machine learning sub-system of the processor ofthe mobile device running in the background can learn the frequentlyused devices of the user. The machine learning sub-system collects userbehavior and maintains a list of frequently used device of the user.This feature is particularly useful when multiple users share the sameapplication account. The machine learning sub-system may be configuredto run in the background to collect user behavior for each of themultiple users and to identify the list of frequently used device foreach user.

In one embodiment, the machine learning sub-system running on thebackground collects event information, such as device name/ID, username/ID, time stamp, temperature, etc. The machine learning sub-systemtrains a machine learning model, such as decision tree, neural network,etc., to learn the unique pattern for each user. Next time, once aparticular user scans his/her face, the machine learning sub-system usesall the current factors as input and estimated the most possible devicesthat the user wants to operate. The machine learning sub-system can alsohave on-line learning capability that continuously learn user's behaviorand improve the estimation result.

Retrieving Connected Devices by Geolocation

Conventional control Apps provides the same user interface for all usersunder all circumstances, regardless of the physical locations of theusers. In reality, location information can be helpful to the user toaccess the most relevant connected devices more efficiently. Inpractice, most of the time, users tend to operate connected devices thatare nearby. In embodiments of the present disclosure, the geolocationinformation is used as a filter to retrieve devices within the vicinityof the user. A positioning sub-system in the processor of the mobiledevice provides geolocation information associated with the user. Insome cases, the geolocation information does not have to be veryaccurate. For the purpose of the geolocation retrieval method, themobile device only needs to provide the geolocation information toindicate which area or which room the user is located.

In embodiments of the present disclosure, the geolocation retrievalmethod can implement various methods of acquiring geolocationinformation. In one example, vision-based methods such as Simultaneouslocalization and mapping (SLAM) allows the system to construct or updatea map of an unknown environment while simultaneously keeping track oflocation within it. In another example, Beacon-based methods use, e.g.Bluetooth low energy (BLE), to detect the proximity of connecteddevices. In another example, sensor-based methods, such as motionsensor, proximity sensor, or Infra-Red sensor, can detect occupancy ofthe room. In another example, Wi-Fi positioning system (WPS) providesgeo-location information by measuring the intensity of the receivedWi-Fi signal from different access points (AP). The geolocationretrieval method can implement any of these aforementioned methods ofacquiring geolocation information or other suitable methods.

FIG. 11 is a flowchart illustrating a geolocation retrieval method insome embodiments. Referring to FIG. 11, a geolocation retrieval method250 may be implemented in a mobile device and executed by the controlApp in embodiments of the present disclosure.

At step 252, the method 250 initiates the control App and determines ageolocation of the mobile device. At step 254, the method 250 accesses adatabase of connected devices. The database of connected devices storesthe geolocation and usage data of each connected device. For example,the usage data may include the frequency of use and the last time anddate the connected device has been assessed. In some embodiments, theusage data and/or the geolocation of each connected device may berecorded for a given user or a given user account. In one example, themethod 250 assesses the database to obtain device usage informationassociated with the determined geolocation. In another example, themethod 250 obtains device usage information associated with a given userat the determined geolocation.

At step 256, the method 250 provides a list of connected devicesassociated with the determined geolocation. For example, the list ofconnected devices includes connected devices that are in the vicinity ofthe determined geolocation.

At step 258, the method 250 receives an input selection selecting aconnected device from the list. At step 260, the method 250 provides auser interface designated for controlling the selected connected device.

FIG. 12 illustrates on example of the operation of the geolocationretrieval method. Referring to FIG. 12, the control App on the mobiledevice determines the geolocation of the user and provides the user witha list of connected devices—a lamp, a television and a game console—onthe control App. In one embodiment, the list of connected devices isprovided on a map and the connected devices are shown at their relativelocation on the map. In this manner, the user can quickly select aconnected device to operate.

In alternate embodiments of the present invention, the image-based userinteraction system and method of the present disclosure can beimplemented in a remote server, such as a cloud server, that isconnected to a mobile device over a data network. The mobile device maybe connected to the remote server through wired or wireless connection.The user uses a mobile device to capture an image and the image istransferred to the cloud server for feature extraction and objectidentification as described above. The remote server provides the userinterface for the identified connected device to the mobile device ofthe user. The implementation of the image-based user interaction systemand method in a mobile device described above is illustrative only andnot intended to be limiting.

The above detailed descriptions are provided to illustrate specificembodiments of the present disclosure and are not intended to belimiting. Numerous modifications and variations within the scope of thepresent disclosure are possible. The present disclosure is defined bythe appended claims.

What is claimed is:
 1. A method in a mobile device for controlling aconnected device, comprising: receiving, at the mobile device, an image;processing, at the mobile device, the image to identify at least a firstimage attribute, the first image attribute comprising an image featureor an image object; accessing, by the mobile device, a database ofconnected devices, each connected device in the database beingidentified by a linked image and being associated with device metadatadescribing at least a device type of the connected device; constructingor updating a map of an environment in the database based on ageolocation of the mobile device and the first image attribute;comparing the first image attribute with the linked images correspondingto the connected devices stored in the database; identifying a subset ofthe connected devices associated with the environment; selecting a firstconnected device among the subset of connected devices; and in responseto the selecting, providing, at the mobile device, a user interfaceconfigured to control the first connected device, the user interfacecomprising a control function corresponding to the device type of thefirst connected device.
 2. The method of claim 1, wherein receiving, atthe mobile device, the image comprises: activating, at the mobiledevice, a control application; and detecting, at the control applicationon the mobile device, the image.
 3. The method of claim 1, wherein:accessing the database of connected devices comprises accessing thedatabase of connected devices including a first set of connecteddevices, each connected device in the first set of connected devicesbeing identified by a linked image of the respective connected deviceitself; and identifying the first connected device comprises identifyingthe first connected device having a linked image matching the firstimage attribute, the first image attribute being associated with thefirst connected device.
 4. The method of claim 1, wherein: accessing thedatabase of connected devices comprises accessing the database ofconnected devices including a second set of dummy devices, each dummydevice in the second set of dummy devices being linked to a respectiveconnected device and being identified by a linked image of an objectthat is not the respective connected device; and identifying the firstconnected device comprises identifying a first dummy device having alinked image matching the first image attribute and retrieving thelinked connected device of the first dummy device as the first connecteddevice.
 5. The method of claim 1, further comprising: activating, at themobile device, a control application; receiving, at the controlapplication on the mobile device, an image of a connected device;processing, at the mobile device, the image to identify at least asecond image attribute, the second image attribute comprising an imagefeature or an image object; receiving, at the mobile device, devicemetadata associated with the connected device; registering, using themobile device, the connected device using the second image attribute asthe linked image and the device metadata; and storing, at the database,the connected device identified by the linked image and being associatedwith the device metadata.
 6. The method of claim 5, further comprising:activating, at the mobile device, a control application; receiving, atthe control application on the mobile device, an image of a dummydevice; processing, at the mobile device, the image to identify at leasta third image attribute, the third image attribute comprising an imagefeature or an image object; receiving, at the mobile device, devicemetadata associated with the dummy device; registering, using the mobiledevice, the dummy device using the third image attribute as the linkedimage and the device metadata; storing, at the database, the dummydevice identified by the linked image and being associated with thedevice metadata. providing the list of connected devices from thedatabase; receiving, at the mobile device, a selection of a connecteddevice; and storing, at the database, linking data of the dummy devicelinking the selected connected device to the dummy device.
 7. The methodof claim 1, wherein accessing the database of connected devicescomprises: storing the database on the mobile device; and accessing thedatabase from the mobile device.
 8. The method of claim 1, whereinaccessing the database of connected devices comprises: storing thedatabase on a hub installed in the environment and associated with oneor more connected devices in the environment; and accessing the databaseon the hub through a local area network.
 9. The method of claim 1,wherein accessing the database of connected devices comprises: storingthe database on a remote server; and accessing the database on theremote server through a data network.
 10. The method of claim 1, whereinprocessing the image to identify at least a first image attributecomprises: processing the image to identify plurality of objects in theimage; and receiving, on the mobile device, a selection of one of theplurality of objects to be processed to identify the first imageattribute.
 11. The method of claim 1, wherein processing the image toidentify at least a first image attribute comprises: processing theimage to identify a foreground object as the first image attribute orprocessing the image to identify a foreground object with a givenbackground image as the first image attribute.
 12. The method of claim1, wherein processing the image to identify at least a first imageattribute comprises processing the image to identify a face image of ahuman as the first image attribute, and the method further comprises:accessing, by the mobile device, a second database of registered users,each user being identified by a linked face image and being associatedwith usage data for one or more connected devices; identifying a firstregistered user having a linked face image matching the face imageidentified in the image; in response to the identifying, providing, atthe mobile device, a second user interface including a list of mostfrequently used connected devices associated with the first registereduser; and in response to receiving an input selecting a second connecteddevice on the second user interface, providing, at the mobile device,the user interface configured to control the second connected device,the user interface comprising a control function corresponding to thedevice type of the second connected device.
 13. The method of claim 1,further comprising: determining, at the mobile device, the geolocationof the mobile device; accessing, by the mobile device, the database ofconnected devices, each connected device being associated with ageolocation and usage data; in response to the determining, providing,at the mobile device, a third user interface including a list ofconnected devices associated with the geolocation; and in response toreceiving an input selecting a third connected device on the third userinterface, providing, at the mobile device, the user interfaceconfigured to control the third connected device, the user interfacecomprising a control function corresponding to the device type of thethird connected device.
 14. The method of claim 13, wherein eachconnected device in the database is further associated with a user andproviding the third user interface including the list of connecteddevices associated with the geolocation comprises: providing the thirduser interface including the list of connected devices associated withthe geolocation and a given user.
 15. A system for controlling aconnected device, comprising: an imaging sensing device configured toreceive an image; a processor; a communication interface; a display; anda memory coupled with the processor, wherein the memory is configured toprovide the processor with instructions which when executed cause theprocessor to: receive an image; process the image to identify at least afirst image attribute, the first image attribute comprising an imagefeature or an image object; access a database of connected devices, eachconnected device in the database being identified by a linked image andbeing associated with device metadata describing at least a device typeof the connected device; construct or update a map of an environment inthe database based on a geolocation of the mobile device and the firstimage attribute; compare the first image attribute with the linkedimages corresponding to the connected devices stored in the database;identify a subset of the connected devices associated with theenvironment; select a first connected device among the subset ofconnected devices and in response to selection of the first connecteddevice, provide a user interface configured to control the firstconnected device, the user interface comprising a control functioncorresponding to the device type of the first connected device.
 16. Thesystem recited in claim 15, wherein the memory is further configured toprovide the processor with instructions which when executed cause theprocessor to: access the database of connected devices including a firstset of connected devices, each connected device in the first set ofconnected devices being identified by a linked image of the respectiveconnected device itself; and identify the first connected device havinga linked image matching the first image attribute, the first imageattribute being associated with the first connected device.
 17. Thesystem recited in claim 15, wherein the memory is further configured toprovide the processor with instructions which when executed cause theprocessor to: access the database of connected devices including asecond set of dummy devices, each dummy device in the second set ofdummy devices being linked to a respective connected device and beingidentified by a linked image of an object that is not the respectiveconnected device; and identify a first dummy device having a linkedimage matching the first image attribute and retrieving the linkedconnected device of the first dummy device as the first connecteddevice.
 18. A method for controlling a connected device, comprising:receiving, at a remote server, an image from a mobile device;processing, at the remote server, the image to identify at least a firstimage attribute, the first image attribute comprising an image featureor an image object; accessing, by the remote server, a database ofconnected devices, each connected device in the database beingidentified by a linked image and being associated with device metadatadescribing at least a device type of the connected device; constructingor updating a map of an environment in the database based on ageolocation of the mobile device and the first image attribute;comparing, at the remote server, the first image attribute with thelinked images corresponding to the connected devices stored in thedatabase; identifying, at the remote server, a subset of the connecteddevices associated with the environment; selecting a first connecteddevice among the subset of connected devices; and in response to theselecting, providing, by the remote server to the mobile device, a userinterface configured to control the first connected device, the userinterface comprising a control function corresponding to the device typeof the first connected device.
 19. The method of claim 18, wherein:accessing the database of connected devices comprises accessing thedatabase of connected devices including a first set of connecteddevices, each connected device in the first set of connected devicesbeing identified by a linked image of the respective connected deviceitself; and identifying the first connected device comprises identifyingthe first connected device having a linked image matching the firstimage attribute, the first image attribute being associated with thefirst connected device.
 20. The method of claim 18, wherein: accessingthe database of connected devices comprises accessing the database ofconnected devices including a second set of dummy devices, each dummydevice in the second set of dummy devices being linked to a respectiveconnected device and being identified by a linked image of an objectthat is not the respective connected device; and identifying the firstconnected device comprises identifying a first dummy device having alinked image matching the first image attribute and retrieving thelinked connected device of the first dummy device as the first connecteddevice.